NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation

Multi-modal MR imaging is routinely used in clinical practice to diagnose and investigate brain tumors by providing rich complementary information. Previous multi-modal MRI segmentation methods usually perform modal fusion concatenating MRIs at an early/middle stage of the network, which hardly explores non-linear dependencies between modalities. In this work, we propose a novel Nested Modality-Aware Transformer (NestedFormer) explicitly explore intra-modality inter-modality relationships for tumor segmentation. Built on transformer-based multi-encoder single-decoder structure, nested high-level representations different modalities apply modality-sensitive gating (MSG) lower scales more effective skip connections. Specifically, conducted our proposed Modality-aware Feature Aggregation (NMaFA) module, enhances long-term within individual via tri-orientated spatial-attention transformer, further complements key contextual information among cross-modality attention transformer. Extensive experiments BraTS2020 benchmark private meningiomas (MeniSeg) dataset show that NestedFormer clearly outperforms state-of-the-arts. The code available https://github.com/920232796/NestedFormer .